The Graphical User Interface (GUI) environment, in the realm of personal computers and workstations, is a radical departure from the traditional Character User Interface (CUI) which has been employed for years in the industry. A Graphical User Interface is a combination of: an all points addressable display such as a cathode-ray tube (CRT), a liquid crystal display (LCD) or a plasma display; input devices such as a mouse, trackball, touch screen, and/or a keyboard; and software with appropriate interfaces which allows a user to control an application program by using stylized screen elements such as menus, windows and controls. The GUI environment is rapidly transforming the shape and nature of the software industry. The active pace of GUI software development is threatened, however, by the fact that applications which are developed for the GUI environment are far more difficult to test than were character application programs. Since GUIs offer a much richer application interface, the size of the testing problem at the user interface level is much greater. For example, GUI applications usually offer a completely new and separate way of invoking program functionality using the mouse, while still maintaining a keyboard interface. This combination at least doubles the size of the testing problem. Both the mouse interface as well as the keyboard interface must be tested for each function allowing/requiring user input. Further, interaction sequences that involve both mouse and keyboard events can be difficult or impossible to replicate with confidence from test to test (e.g., Did the mouse click come before the typed text, or the typed text before the mouse click? Was the mouse moved first? Where exactly?, etc.).
In the traditional CUI environment, an application owned the screen. Not having to contend with other applications, a CUI application could write to specific screen locations without having to fear corrupting or interfering with other applications. The "screen certainty" offered by the CUI world was a great benefit to testers, who have traditionally required repeatability and consistency between executions of an application. But the GUI environment has all but eliminated "screen certainty". There are no fixed screen reference points for a GUI application. An application may appear anywhere on the screen as the GUI decrees, and the user may further move or resize it, in order to accommodate other applications on the same screen. The GUIs have made it quite impossible for Quality Assurance to use traditional testing techniques on any but a tiny fraction of the permutations an application may experience in the hands of a user.
A closer examination of the operation of a GUI application reveals that testing such an application is even more complex than is facially apparent. Underneath the interface, at the code level, a GUI programmer's difficulty in dealing with the user interface is tremendously more complex than in the traditional CUI environment (which equates to a correspondingly difficult task of testing the interface). In the CUI environment, the application programmer told the user what to do, prompting for input when the program was ready for it, and not before. In the GUI environment, the tables have been turned, with the user generating a series of mouse and keyboard events, to which the program must respond. As a result, it is much harder for application programmers to visualize a GUI application's flow of control. In addition, global and static variables must be used to preserve program state across event boundaries, and such variables are prey to a host of subtle interactional problems. These factors have contributed to real-time programming's perceived complexity for years. With the advent of the GUI environment, these problems have now entered the world of everyday application programming.
A test tool typically uses "scripts" to "drive" an application to one or more states, then "validates" those states for correctness using "baseline" representations of correctness in order to determine whether new versions of software have "regressed" in quality (the latter is known as "regression" testing). The CUI test tool paradigm calls for the use of software event recordings to drive an application to specific states, and bitmap comparisons to validate software state as reflected on the screen. The two primary components of the CUI test tool paradigm, software event recording and bitmap validation, are wholly inadequate in the GUI environment.
Prior art test tools (CUI test tools) performed software recording by detecting and storing low-level events, such as mouse moves, clicks, drags, and keyboard events. The goal of the recording is to allow a test script to drive the target software by replaying a recording of a live interaction with the software being tested. Software recording was feasible in CUI environments, because the target application owned the screen and stayed in one place throughout user sessions. Software event recordings work by preserving the physical screen locations ("hard-coded") of the events they record. A typical software recording would look something like the following:
Move 110,30 PA1 Move 125,20 PA1 ButtonPress 1 PA1 ButtonRelease 1 PA1 Move 125,60 PA1 Move 125,70 PA1 ButtonPress 1 PA1 ButtonRelease 1
The above test calls for dropping down the File menu and clicking on the Open selection (although the actions cannot be discerned from the above software recording). The numeric values after the MOVE commands are hard-coded screen locations for the objects under test (the objects being the File menu and the Open option on the File menu). The value of such recordings is minimal in the GUI environment since there is no longer any certainty with respect to the physical screen location of such objects. In fact, part of the appeal and power of the GUI environment is the flexibility and mutability of the physical screen interface.
The basic policy of the GUI environment holds that no assumptions should be made about the location and size of objects on the screen. Instead, an interested party (e.g., an application program) must ask the GUI for this information, which is only valid at that moment, and can't be usefully recorded for playback at a later time. A software recording at the GUI event level is a long list of assumptions about the state of the screen that were true at the time the recording was made, but are not necessarily true at playback time. In the software recording example described above, the user might have moved the location of the File menu and, therefore, the hard-coded screen locations for the objects under test are no longer valid, and the entire test case is invalid. And yet, the prior art method of record and playback of events against hard-coded screen locations is the primary test mechanism offered to GUI testers today.
An additional drawback of software recording involves a "Catch-22". The software under test must be correctly functioning in order to record a test that will determine whether the software is functioning correctly. An automated regression test suite cannot be built until the software is nearly working and until there aren't going to be even minor changes to the user interface.
The other primary prior art tool available to GUI testers is bitmap validation. Bitmap validation is the ideological fellow traveller of software recording. Using a "map" of the physical screen bits, the bitmap validation process compares a current state of the screen with a previously stored representation of what was adjudged to be a "correct" screen state. But as with software event recording, the nature of the GUI environment renders this tool all but useless. The size and screen position of a GUI application's windows naturally tend to be different from test run to test run. A GUI application's state is sensitively dependent on initial conditions. A small initial difference can produce a substantial change later on. For example, Microsoft Windows.TM. (Microsoft Windows is a trademark of the Microsoft Corporation) tiles applications as they first come up in specific screen positions, unless an application overrides the GUI. The starting position of an application during a given test run will depend on how many applications were brought up since the system was last booted. So, screen bitmaps are profoundly sensitive to exactly the wrong things from the perspective of GUI application testers.
One other significant aspect of application testing is the growing trend towards cross-platform application development. Cross-platform as used here, means pertaining to more than one GUI. Cross-platform testing, therefore, is the testing of an application that runs on several platforms (e.g., an application which runs under both Motif.TM. (Motif is a trademark of the Open Software Foundation) and Microsoft Windows. The term cross-platform can also refer to the hardware/Operating System (OS) level in the sense that Motif.TM., for example, is a cross-platform GUI because it runs on many hardware/OS combinations. As an application becomes popular on one platform, there are tremendous financial incentives for the application developer to "port" the application to a different platform. Unfortunately, the traditional CUI test suites developed for one platform are invariably invalid for testing on a different platform. This means that the entire test suite for the application will have to be entirely regenerated for each new platform. This is particularly difficult for cross-platform development since the "porting" of an a plication can proceed rather rapidly, and the majority of the work and pressure is on the tester to validate the porting process.
It is, therefore, one object of the invention to perform automated testing of an application program's operation with Graphical User Interface (GUI).
It is also an object of the invention to perform testing against logical screen objects and not hardcoded screen coordinates.
It is a further object of the invention tool to support the creation of test suites that are easily portable to other GUIs.
It is another object of the invention to be able to create a test suite for an application without having to wait until the application is completely developed.